1. Field of the Invention
The present invention relates to searches and more specifically to managing resources to process data centric searches.
2. Introduction
Data centric searches have grown in popularity in the last 15 years. Companies like Yahoo, AltaVista, Ask.com, and Google have risen by providing search services over the World Wide Web. Typically users visit a web page and search for items they wish to find by entering search terms in a text box on a web page. A search engine processes those queries in a large data center containing hundreds, thousands, or even hundreds of thousands of individual computers networked together as a grid or a cluster as a compute environment. A search engine typically distributes the search database across many computers. However, when an event occurs which piques the interest of many Internet users, requests for information about the event can overwhelm search engines in a short period of time. For example, the Beijing 2008 Olympics was an event that drove an enormous amount of web searches, most of which were substantially the same or contained a very similar element. In a typical search engine data center, that means that one computer or a fixed set of computers containing highly sought-after information are repeatedly queried. Such similar queries may be considered within the same domain. The sudden workload increase on that computer or set of computers often leads to a decrease in quality of service. For example, where a server may service a query in 0.02 seconds under normal conditions, the same server may service a query in 0.08 seconds or more under extreme load.
While the Olympics are a predictable example of an event that leads to many spikes in searches, other events are less predictable, for example natural disasters such as hurricanes and earthquakes and unnatural disasters such as political scandals. Whereas searches related to the Olympics are spread out over two weeks or more and search volume gradually increases, other events rapidly spike from being statistically insignificant to occupying a substantial percentage of overall searches in a very short period of time.
In many cases, these high-volume, event-driven searches are the highest value searches (i.e. the most important to users), but because of the close temporal proximity and high volume of requests, the individual servers in the compute environment which contain the necessary data are the least responsive. The net result is that the most important searches receive the worst service.
Data searches as used herein may apply to any kind of data search or data centric transaction. Web searches may involve a search such as a Google search in which the data is indexed data owned by Google and obtained via web crawling algorithms over the Internet. The data searched may be webpages themselves such as where many users access the same webpage like the drudgereport or the CNN websites.
Accordingly, what is needed in the art is an improved way to manage searches such that the most frequent searches are serviced in a timely manner.